add distributed locking to jobs in alerting #1403

sbcd90 · 2024-02-03T04:04:43Z

Issue #, if available:

Description of changes:
Alerting uses scheduled jobs to trigger monitor runs. In case a monitor run is a long running job, the job may not get completed before the next job is scheduled to run. This introduces a scenario where multiple jobs run in parallel resulting a race condition to acquire resources.
This pr introduces distributed locking to scheduled jobs in alerting. It ensures only 1 active job can run at an instant of time.

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Subhobrata Dey <[email protected]>

engechas · 2024-03-08T20:59:01Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorRunnerService.kt

+                        } ?: return@launch
+                        logger.debug("lock ${lock!!.lockId} acquired")
+                        logger.debug(
+                            "PERF_DEBUG: executing workflow ${job.id} on node " +


Minor: I think we should get rid of the PERF_DEBUG prefixes in these log statements. I'm fine if we fix it later though

engechas · 2024-03-08T21:00:34Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorRunnerService.kt

-                            monitorCtx.clusterService!!.state().nodes().localNode.id
-                    )
-                    runJob(job, periodStart, periodEnd, false)
+                    var lock: LockModel? = null


Minor: this code looks identical to 303-317. We should create a private method to avoid the duplication

engechas · 2024-03-08T21:08:59Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+                                    override fun onResponse(existingLock: LockModel?) {
+                                        if (existingLock != null) {
+                                            if (isLockReleased(existingLock)) {
+                                                log.debug("lock is released or expired: {}", existingLock)


Non-blocking - this line is 10x nested. We should avoid deeply nesting code as it hurts readability. We can break much of this into private methods to avoid the nesting

engechas · 2024-03-08T21:15:41Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorRunnerService.kt

+                    try {
+                        lock = monitorCtx.client!!.suspendUntil<Client, LockModel?> {
+                            monitorCtx.lockService!!.acquireLock(job, it)
+                        } ?: return@launch


We could consider adding adding some delay between retries on acquiring the lock

eirsep · 2024-03-08T21:36:37Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+private val log = LogManager.getLogger(LockService::class.java)
+
+class LockService(private val client: Client, private val clusterService: ClusterService) {
+    private var testInstant: Instant? = null


why is this called testInstant??

eirsep · 2024-03-08T21:37:36Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+                    }
+
+                    override fun onFailure(ex: Exception) {
+                        log.error("Failed to update config index schema", ex)


log error only in else condition

eirsep · 2024-03-08T21:38:16Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+            listener.onResponse(true)
+        } else {
+            val indexRequest = CreateIndexRequest(LOCK_INDEX_NAME).mapping(lockMapping())
+                .settings(Settings.builder().put("index.hidden", true).build())


plz add primary and replica settings based on our general practices for system index.

eirsep · 2024-03-09T00:31:54Z

alerting/src/main/kotlin/org/opensearch/alerting/service/DeleteMonitorService.kt

@@ -148,6 +154,10 @@ object DeleteMonitorService :
        }
    }

+    private suspend fun deleteLock(monitor: Monitor) {
+        client.suspendUntil<Client, Boolean> { lockService.deleteLock(LockModel.generateLockId(monitor.id), it) }


eirsep · 2024-03-09T02:02:02Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockModel.kt

+    )
+
+    override fun toXContent(builder: XContentBuilder, params: ToXContent.Params): XContentBuilder {
+        builder.startObject()


lockid missing?

eirsep · 2024-03-09T02:02:13Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockModel.kt

+    }
+
+    companion object {
+        const val SCHEDULED_JOB_ID = "scheduled_job_id"


lockid missing?

eirsep · 2024-03-09T02:04:08Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+    ) {
+        val lockId = LockModel.generateLockId(scheduledJobId)
+        createLockIndex(
+            object : ActionListener<Boolean> {


plz use actionlistener.wrap()

eirsep · 2024-03-09T02:04:28Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+                            listener.onResponse(null)
+                        }
+                    } else {
+                        listener.onResponse(null)


what does null mean? and what does failure mean?

eirsep · 2024-03-09T02:05:40Z

core/src/main/kotlin/org/opensearch/alerting/core/lock/LockService.kt

+                            listener.onResponse(LockModel.parse(parser, response.seqNo, response.primaryTerm))
+                        } catch (e: IOException) {
+                            log.error("IOException occurred finding lock", e)
+                            listener.onResponse(null)


Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit bcaa4e3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Signed-off-by: Subhobrata Dey <[email protected]>

…ject#1307) * Added 2.11.1 release notes. * Added 2.11.1 release notes. --------- (cherry picked from commit 06c1b8a) Signed-off-by: AWSHurneyt <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> fix workflow security tests in alerting (opensearch-project#1310) (opensearch-project#1311) Signed-off-by: Subhobrata Dey <[email protected]> Increment version to 2.12.0-SNAPSHOT (opensearch-project#1239) Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> [Backport 2.x] Reference get monitor and search monitor action / request / responses from common-utils (opensearch-project#1315) * Use get monitor action / req / resp from common-utils Signed-off-by: Tyler Ohlsen <[email protected]> * Dummy commit to retrigger Signed-off-by: Tyler Ohlsen <[email protected]> --------- Signed-off-by: Tyler Ohlsen <[email protected]> optimize doc-level monitor execution workflow for datastreams (opensearch-project#1302) (opensearch-project#1322) Signed-off-by: Subhobrata Dey <[email protected]> Update to Gradle 8.5 (opensearch-project#1369) (opensearch-project#1371) Signed-off-by: Andriy Redko <[email protected]> [Backport 2.x] Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) (opensearch-project#1384) * Inject namedWriteableRegistry during ser/deser of SearchMonitorAction (opensearch-project#1382) Signed-off-by: Tyler Ohlsen <[email protected]> * remove bin files Signed-off-by: Tyler Ohlsen <[email protected]> * remove core bin Signed-off-by: Tyler Ohlsen <[email protected]> --------- Signed-off-by: Tyler Ohlsen <[email protected]> Don't attempt to parse workflow if it doesn't exist (opensearch-project#1346) (opensearch-project#1359) (cherry picked from commit 733fd4e) Signed-off-by: Chase Engelbrecht <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Set docData to empty string if actual is null (opensearch-project#1325) (opensearch-project#1334) (cherry picked from commit 008e076) Signed-off-by: Chase Engelbrecht <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> removed default admin credentials for alerting (opensearch-project#1399) (opensearch-project#1400) (cherry picked from commit 3c50f7d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Dennis Toepker <[email protected]> ipaddress lib upgrade as part of cve fix (opensearch-project#1397) (opensearch-project#1407) (cherry picked from commit 8d59060) Signed-off-by: Riya Saxena <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Bulk index findings and sequentially invoke auto-correlations (opensearch-project#1355) (opensearch-project#1410) * Bulk index findings and sequentially invoke auto-correlations * Bulk index findings in batches of 10000 and make it configurable * Addressing review comments * Add integ tests to test bulk index findings * Fix ktlint formatting --------- (cherry picked from commit b561965) Signed-off-by: Megha Goyal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Add 2.12 release notes (opensearch-project#1408) (opensearch-project#1413) * Add 2.12 release notes * Fix release notes PR * Add 2 more PRs --------- (cherry picked from commit b10eaad) Signed-off-by: Chase Engelbrecht <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> [Backport 2.x] Implemented cross-cluster monitor support (opensearch-project#1404) (opensearch-project#1412) * Implemented cross-cluster monitor support (opensearch-project#1404) * Updated alert mappings to accommodate cross-cluster cluster metrics monitors. Signed-off-by: AWSHurneyt <[email protected]> * Implemented support for cross-cluster cluster metrics monitors. Implemented GetRemoteIndexes API to populate the frontend UI with details regarding the remote clusters, and indexes. Signed-off-by: AWSHurneyt <[email protected]> * Fixed a writeable test after changing QueryLevelTriggerRunResult from a data class to an open class for inheritability. Signed-off-by: AWSHurneyt <[email protected]> * Fixed ktlint errors. Signed-off-by: AWSHurneyt <[email protected]> * Removed changes to IndexUtils as they're only needed by doc monitors. Signed-off-by: AWSHurneyt <[email protected]> --------- Signed-off-by: AWSHurneyt <[email protected]> (cherry picked from commit ea36996) Signed-off-by: AWSHurneyt <[email protected]> * Fixed a test. Signed-off-by: AWSHurneyt <[email protected]> --------- Signed-off-by: AWSHurneyt <[email protected]> Add publishToMavenLocal in build.sh (opensearch-project#1418) (opensearch-project#1419) (cherry picked from commit 4cdc1d1) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (opensearch-project#1432) (opensearch-project#1434) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> bacport PRs opensearch-project#1445, opensearch-project#1430, opensearch-project#1441, opensearch-project#1435 to 2.x (opensearch-project#1452) * Add jvm aware setting and max num docs settings for batching docs for percolate queries (opensearch-project#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <[email protected]> * fix stats logging Signed-off-by: Surya Sashank Nistala <[email protected]> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (opensearch-project#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test for settings check Signed-off-by: Surya Sashank Nistala <[email protected]> * fix ktlint Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * clean up doc level queries on dry run (opensearch-project#1430) Signed-off-by: Joanne Wang <[email protected]> * optimize sequence number calculation and reduce search requests in doc level monitor execution (opensearch-project#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <[email protected]> * fix tests Signed-off-by: Surya Sashank Nistala <[email protected]> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test Signed-off-by: Surya Sashank Nistala <[email protected]> * add javadoc Signed-off-by: Surya Sashank Nistala <[email protected]> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Joanne Wang <[email protected]> Co-authored-by: Joanne Wang <[email protected]> [Backport 2.x] Add an _exists_ check to document level monitor queries (opensearch-project#1425) (opensearch-project#1456) * Add an _exists_ check to document level monitor queries (opensearch-project#1425) * clean up and add integ tests Signed-off-by: Joanne Wang <[email protected]> * refactored out common method and renamed test Signed-off-by: Joanne Wang <[email protected]> * remove _exists_ flag Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> * fix integ test Signed-off-by: Joanne Wang <[email protected]> --------- Signed-off-by: Joanne Wang <[email protected]> add distributed locking to jobs in alerting (opensearch-project#1403) (opensearch-project#1458) Signed-off-by: Subhobrata Dey <[email protected]>

Signed-off-by: Subhobrata Dey <[email protected]>

…, #1441 to 2.9 (#1469) * optimize doc-level monitor execution workflow for datastreams (#1302) * optimize doc-level monitor execution for datastreams Signed-off-by: Subhobrata Dey <[email protected]> * add more tests to address comments Signed-off-by: Subhobrata Dey <[email protected]> * add integTest for multiple datastreams inside a single index pattern * add integTest for multiple datastreams inside a single index pattern Signed-off-by: Subhobrata Dey <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> * Bulk index findings and sequentially invoke auto-correlations (#1355) * Bulk index findings and sequentially invoke auto-correlations Signed-off-by: Megha Goyal <[email protected]> * Bulk index findings in batches of 10000 and make it configurable Signed-off-by: Megha Goyal <[email protected]> * Addressing review comments Signed-off-by: Megha Goyal <[email protected]> * Add integ tests to test bulk index findings Signed-off-by: Megha Goyal <[email protected]> * Fix ktlint formatting Signed-off-by: Megha Goyal <[email protected]> --------- Signed-off-by: Megha Goyal <[email protected]> * fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Add jvm aware setting and max num docs settings for batching docs for percolate queries (#1435) * add jvm aware and max docs settings for batching docs for percolate queries Signed-off-by: Surya Sashank Nistala <[email protected]> * fix stats logging Signed-off-by: Surya Sashank Nistala <[email protected]> * add queryfieldnames field in findings mapping Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * clean up doc level queries on dry run (#1430) Signed-off-by: Joanne Wang <[email protected]> * optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc (#1441) * optimize to fetch only fields relevant to doc level queries in doc level monitor Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test for settings check Signed-off-by: Surya Sashank Nistala <[email protected]> * fix ktlint Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445) * optimize sequence number calculation and reduce search requests by n where n is number of shards being queried in the executino Signed-off-by: Surya Sashank Nistala <[email protected]> * fix tests Signed-off-by: Surya Sashank Nistala <[email protected]> * optimize check indices and execute to query only write index of aliases and datastreams during monitor creation Signed-off-by: Surya Sashank Nistala <[email protected]> * fix test Signed-off-by: Surya Sashank Nistala <[email protected]> * add javadoc Signed-off-by: Surya Sashank Nistala <[email protected]> * add tests to verify seq_no calculation Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Surya Sashank Nistala <[email protected]> * add distributed locking to jobs in alerting (#1403) Signed-off-by: Subhobrata Dey <[email protected]> * 2.9.1 version bump Signed-off-by: Surya Sashank Nistala <[email protected]> * fix compilation issues Signed-off-by: Surya Sashank Nistala <[email protected]> * dummy commit Signed-off-by: Surya Sashank Nistala <[email protected]> * fix findings index schema version tsts Signed-off-by: Surya Sashank Nistala <[email protected]> --------- Signed-off-by: Subhobrata Dey <[email protected]> Signed-off-by: Megha Goyal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Surya Sashank Nistala <[email protected]> Signed-off-by: Joanne Wang <[email protected]> Co-authored-by: Subhobrata Dey <[email protected]> Co-authored-by: Megha Goyal <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Joanne Wang <[email protected]>

opensearch-trigger-bot · 2024-05-02T13:40:59Z

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.11
# Create a new branch
git switch --create backport-1403-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bcaa4e36d86675850b537425b066f3c15cda15cc
# Push it to GitHub
git push --set-upstream origin backport-1403-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport-1403-to-2.11.

sbcd90 requested review from lezzago, AWSHurneyt, eirsep, getsaurabh02, praveensameneni, qreshi, bowenlan-amzn and rishabhmaurya as code owners February 3, 2024 04:04

sbcd90 marked this pull request as draft February 3, 2024 04:04

sbcd90 force-pushed the dist_lock branch 4 times, most recently from 47a4888 to e2a52df Compare February 6, 2024 08:52

sbcd90 force-pushed the dist_lock branch 2 times, most recently from aa8eef4 to 7c90068 Compare March 8, 2024 20:52

sbcd90 marked this pull request as ready for review March 8, 2024 20:53

sbcd90 requested review from engechas and riysaxen-amzn as code owners March 8, 2024 20:53

add distributed locking to jobs in alerting

b54b5e7

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 force-pushed the dist_lock branch from 7c90068 to b54b5e7 Compare March 8, 2024 20:55

engechas approved these changes Mar 8, 2024

View reviewed changes

eirsep reviewed Mar 8, 2024

View reviewed changes

eirsep reviewed Mar 9, 2024

View reviewed changes

eirsep approved these changes Mar 9, 2024

View reviewed changes

eirsep merged commit bcaa4e3 into opensearch-project:main Mar 9, 2024
7 of 18 checks passed

eirsep added the backport 2.x label Mar 9, 2024

opensearch-trigger-bot bot mentioned this pull request Mar 9, 2024

[Backport 2.x] add distributed locking to jobs in alerting #1458

Merged

sbcd90 mentioned this pull request Mar 9, 2024

address review comments for #1403 #1459

Closed

sbcd90 pushed a commit that referenced this pull request Mar 9, 2024

add distributed locking to jobs in alerting (#1403) (#1458)

7ad2209

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 13, 2024

add distributed locking to jobs in alerting (opensearch-project#1403)

ae97006

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep mentioned this pull request Mar 13, 2024

[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9 #1469

Merged

eirsep pushed a commit to eirsep/alerting that referenced this pull request Mar 14, 2024

add distributed locking to jobs in alerting (opensearch-project#1403)

78955a6

Signed-off-by: Subhobrata Dey <[email protected]>

jowg-amazon added the backport 2.11 label May 2, 2024

opensearch-trigger-bot bot added the failed backport label May 2, 2024

sbcd90 mentioned this pull request May 2, 2024

add distributed locking to jobs in alerting #1542

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add distributed locking to jobs in alerting #1403

add distributed locking to jobs in alerting #1403

sbcd90 commented Feb 3, 2024 •

edited

Loading

engechas Mar 8, 2024

eirsep Mar 9, 2024

engechas Mar 8, 2024

engechas Mar 8, 2024

engechas Mar 8, 2024

eirsep Mar 8, 2024

eirsep Mar 8, 2024

eirsep Mar 8, 2024

eirsep Mar 9, 2024

eirsep Mar 9, 2024

eirsep Mar 9, 2024

eirsep Mar 9, 2024

eirsep Mar 9, 2024

eirsep Mar 9, 2024

opensearch-trigger-bot bot commented May 2, 2024

add distributed locking to jobs in alerting #1403

add distributed locking to jobs in alerting #1403

Conversation

sbcd90 commented Feb 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented May 2, 2024

sbcd90 commented Feb 3, 2024 •

edited

Loading